Global Optimization issues in Supervised Learning. An overview

The paper presents an overview of global issues in optimization methods for Supervised Learning (SL). We focus on Feedforward Neural Networks with the aim of reviewing global methods specifically devised for the class of continuous unconstrained optimization problems arising both in Multi Layer Perceptron/Deep Networks and in Radial Basis Networks. We first recall the learning optimization paradigm for FNN and we briefly discuss global scheme for the joined choice of the network topologies and of the network parameters. The main part of the paper focus on the core subproblem which is the unconstrained regularized weight optimization problem. We review some recent results on the existence of local-non global solutions of the unconstrained nonlinear problem and the role of determining a global solution in a Machine Learning paradigm. Local algorithms that are widespread used to solve the continuous unconstrained problems are addressed with focus on possible improvements to exploit the global properties. Hybrid global methods specifically devised for SL optimization problems which embed local algorithms are discussed at the end.